1. Field of the Invention
The present invention relates to digital image processing and, more particularly, to white-point estimation for color digital images.
2. Description of the Related Art
Under a large variety of scene illuminants, a human observer sees the same range of colors; a white piece of paper remains resolutely white independent of the color of light under which it is viewed. In contrast, color imaging systems (e.g., digital cameras) are less color constant in that they will often infer the color of the scene illuminant incorrectly. Unless the color constancy problem is solved, color appearance models cannot be used to guide image processing, and such processing is necessary for accurate (and acceptable) color reproduction.
In our work on testing color appearance models we have found that several of the color appearance models perform well when asked to compensate for a range of illuminants. See, P. M. Hubel and G. D. Finlayson, "Sharp Transforms for Color Appearance", Accepted for publication and presentation at SPIE Electronic Imaging Conference: Device Independent Color, San Jose, Calif., (1998). The main factor prohibiting the use of such models in digital photography (and probably most other applications) is the requirement that the color of the scene illumination must be known. In most situations, we simply do not have this information.
In processing the digital camera image, we must either measure the color of the scene illumination or estimate its color from the image data. Of course, in working on digital imaging systems, it is not practical to have an illumination sensor and expect users to calibrate to a white reference. If biological imaging systems achieve color constancy without an illumination color sensor, then it should be possible for us to achieve color constancy from just the image data (otherwise we would have evolved with spectrophotometers and white reference tiles mounted on our foreheads!).
Many solutions have been proposed for the white-point estimation problem. Land, Buchsbaum, Gershon, and others, proposed that the average color of a scene is gray and so the white-point chromaticity corresponds to the average image chromaticity (we refer to this method as Gray World). See, E. H. Land, "Recent Advances in Retinex Theory", Vision Research, 26, p. 7-21, (1986); G. Buchsbaum, "A Spatial Processor Model for Object Color Perception", Journal of the Franklin Institute 310, p. 1-26 (1980); and, R. Gershon, A. D. Jepson and J. K. Tsotsos, "From [R,G,B] to Surface Reflectance: Computing Color Constant Descriptors in Images", Perception, p. 755-758 (1988). Land proposed that the maximum pixel responses, calculated in the red, green, and blue color channels individually, can also be used as a white-point estimate (we refer to this method as Max.RGB). See, E. H. Land, "The Retinex Theory of Color Vision," Scientific American, p. 108-129, (1977).
Maloney, Wandell, D'Zmura, Iverson, Funt, Drew, and others, have formulated the white-point estimation problem as an equation-solving exercise. See, L. T. Maloney and B. A. Wandell, "Color Constancy: a Method for Recovering Surface Spectral Reflectance", J. Opt. Soc. Am. A, p. 29-33, (1986); B. A. Wandell, "The Synthesis and Analysis of Color Images", IEEE Trans. Patt. Anal. and Mach. Intell. PAMI-9, p. 2-13, (1987); M. D'Zmura, "Color Constancy: Surface Color from Changing Illumination", J. Opt. Soc. Am. A (9), p. 490-493 (1992); M. D'Zmura and G. Iverson, "Color Constancy. I. Basic Theory of Two-stage Linear Recovery of Spectral Descriptions for Lights and Surfaces", J. Opt. Soc. Am. A (10), p. 2148-2165 (1993); and, B. V. Funt and M. S. Drew, "Color Constancy Computation in Near-Mondrian Scenes Using a Finite Dimensional Linear Model", IEEE Computer Vision and Pattern Recognition Proceedings, p. 544-549, (1988).
In contrast, Tominaga, Wandell, Funt, Tsukada, Drew, and others, have shown that in principle the white-point might be found by exploiting the physics of the world, for example by finding specularity or interreflection regions in images. See, S. Tominaga and B.A. Wandell, "Standard Surface-Reflectance Model and Illuminant Estimation", J. Opt. Soc. Am. A, (6), p. 576-584, (1989); B. V. Funt, M. S. Drew and J. Ho, "Color Constancy from Mutual Reflection", IJCV, (6), p. 5-24 (1991); M. Tsukada and Y. Ohta, "An Approach to Color Constancy Using Multiple Images", IEEE Comp. Soc, (1990); and, M. S. Drew, "Optimization Approach to Dichromatic Images", Journal of Mathematical Imaging, (1993).
All these methods are similar, however, in the respect that they afford poor estimation performance. See, D. A. Brainard and B. A. Wandell, "Analysis of the Retinex Theory of Color Vision", J. Opt. Soc. Am. A (36), p. 1651-1661, (1986); G. D. Finlayson, "Color Constancy in Diagonal Chromaticity Space", IEEE Computer Society: Proccedings of the fifth International Conference on Computer, p. 218-223, (1995); and, K. Barnard, "Computational Color Constancy: Taking Theory into Practice", MSc thesis, Simon Fraser University, School of Computing (1995). These methods fail because they make assumptions about images which do not in general hold: the average of a scene is not always gray, and specularities may or may not appear in images (and when specularities do appear they are not easily found). Each of the methods is easily discredited.
That these methods fail has inspired other authors to search for color constancy algorithms which are based only on weak (that is reasonable) scene assumptions. Forsyth developed a theory of estimation based solely on the observation that the range of colors measured by a camera (or the eye) depends on the color of the light (the reddest red color cannot occur under the bluest light). See, D. Forsyth, "A Novel Algorithm for Color Constancy", IJVC (5), p. 5-36, (1990). This idea was refined by Finlayson (the Color in Perspective method) who observed that illuminant color is itself quite restricted. See, G. D. Finlayson and S. Hordley, "Selection for Gamut Mapping for Color Constancy", Proceedings of the British Machine Vision Conference, to appear (1997).
The invention described here is an improvement on an earlier technique to determine the color of illumination in a scene. See, G. D. Finlayson, "Color in Perspective", IEEE Transactions on Pattern Analysis and Machine Intelligence, p. 1034-1038, (1996). Color by Correlation not only performs significantly better than other methods but is a simple, elegant solution to a problem that has eluded scientists working on color for over a century. See, e.g., H. von Helmholtz Handbuch der Physiologischen Optik (2nd ed.), Voss, Hamburg, (1896).
Because the Color in Perspective method is the closest precursor to the correlation method presented here, it is worth reviewing the details of how it works. In a preprocessing stage, Color in Perspective calculates models of plausible surface colors and plausible illuminant colors. These correspond to bounded regions of chromaticity space. A chromaticity image, of many surfaces viewed under a single scene illuminant, must be simultaneously consistent with both these constraint sets. That is, solving for color constancy amounts to a constraint satisfaction task; the output of Color in Perspective is the set of possible estimates of the white-point in an image. The mathematics of how Color in Perspective solves the constraint task is somewhat laborious (it involves calculating and intersecting many convex constraint sets). In addition, the method is highly sensitive to spurious inconsistencies. For example the presence of an aperture color in an image can force the solution set to be empty. The correlation method presented in this paper can be used to calculate the Color in Perspective constraint set. However, the new method is very much simpler (faster!) and is also more robust (is not sensitive to spurious outliers).
Adopting only the weak assumptions made in the Forsyth and Finlayson methods makes it impossible to return a unique estimate of the white point. Rather, a range of possible answers is returned, any one of which might be possible. Of course a single estimate must still be chosen from this set, and a variety of estimators have in fact been proposed for this task. Forsyth suggests that after white-balancing (discounting any color biases due to illumination), the image colors should be as colorful as possible. Finlayson and Hordley (repeat-G. D. Finlayson and S. Hordley, "Selection for Gamut Mapping for Color Constancy", Proceedings of the British Machine Vision Conference, to appear (1997)) propose the mean as a more robust estimate, and D'Zmura and Iverson (and Brainard and Freeman) suggest a maximum likelihood estimation. See, M. D'Zmura, G. Iverson and B. Singer, "Probabilistic Color Constancy", in Geometric Representations of Perceptual Phenomena: Papers in Honor of Tarow Indow's 70th birthday, ed. R. D. Luce et al, Lawrence Erlbaum Associates, (1994); and, D. H. Brainard and W. T. Freeman, "Bayesian Method for Recovering Surface and Illuminant Properties from Photosensor Responses", in Human Vision, Visual Processing, and Digital Display V, p. 364 (1994). The latter estimator is particularly relevant to this work because our proposed solution can, as a special case, also support the maximum likelihood case. However, unlike the D'Zmura and Iverson method, our solution is computationally simple. Our method is so simple that maximum likelihood estimation can be provided at video frame rate.
The key observation that we exploit in our method is that the number of colors, and the range of white-points that a camera can sense, is finite. That is the white-point estimation is an intrinsically discrete problem. Funt et al. recently proposed white-point estimation as a discrete neural computation problem. See, B. V. Funt, V. Cardei and K. Barnard, "Learning Color Constancy", 4th IS&T and SID Color Imaging (1996). Here, image chromaticities are fed into a "trained" neural network which then returns a white-point estimate as output. Unfortunately, this method works as a "black-box" and so one cannot say too much about the estimates that are made, such as the estimate confidence. Moreover, physically impossible estimates can also be made.
Thus, it can be seen that illuminant estimation techniques impose color correction, processing speed, robustness and image quality limits upon color digital imaging processing devices, and hinder the use of these devices in many applications.
Therefore, there is an unresolved need for an illuminant estimation technique that can improve color digital imaging quality by quickly, accurately and robustly estimating illuminant information associated with a color digital image in order to correct color of the image based on the illuminant.